feat(dynacell): controlled FOV-parallel eval + thread budgeting - #426

…(C2)

Adds always-on region timers (RegionTimer context manager) that record
wall time of FOV-level and per-T regions to <save_dir>/eval_timing.csv.
Region taxonomy matches actual execution placement at pipeline.py:341-505:

  FOV-level (t=None): mask_gt, mask_pred, cp_gt, deep_gt_dinov3,
  deep_gt_dynaclr, deep_gt_celldino, features_pred_per_t, microssim, seg_write
  Per-T: pixel_metrics, mask_metrics, feature_pairwise

Timing overhead is ~0.1 ms per region per timepoint; ~120 ms total on a
9-h eval (~4e-6 of wall time) — no gating knob.

Per-T memory hygiene knobs (runtime.cuda_empty_cache_every_n_timepoints,
runtime.gc_collect_every_n_fovs) ship at 0 (no behavior change) in this
commit. The plan calls for measuring and flipping defaults in C2 — that
measurement requires a real eval run on the test48 zarr that exceeds this
session's scope. Defaults stay at 0; observed late-T degradation in
prior A549 ER evals (18 -> 61 s/T over 10 timepoints) is the documented
motivation but not the measured fix in this commit.

DYNACELL_FORCE_PER_T_HYGIENE=1 env var flips both knobs on at runtime
regardless of YAML defaults — operator escape hatch for post-ship
mitigation without a code change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…esult (C3)

Pure refactor: extract the per-FOV body of evaluate_predictions into a
module-level function _process_one_fov(...) -> FovResult and an
_aggregate_fov_result(...) parent-side aggregator. Serial-path behavior
is identical to the prior code (verified: 384/48 pass/fail parity with
HEAD~2; the 48 failures are all pre-existing).

FovResult carries everything one FOV contributes to the run as a
picklable dataclass:
- per_t_pixel_rows (with microssim already merged in per pipeline.py:471
  semantics — no separate microssim_rows field),
- per_t_mask_rows, per_t_feature_rows,
- seg_array of shape (T, 2, D, H, W) bool (channel 0 = pred, channel 1 = GT),
- 4 _BackboneLists for cp/dinov3/dynaclr/celldino — each holds the per-T
  pred_feats/gt_feats/pred_fovs/gt_fovs/pred_ts/gt_ts arrays. Workers
  preserve the existing `if size > 0: append` guard so empty per-T slots
  never enter the lists; aggregator extends parent lists unconditionally.
- timings slice from the per-process region_timer collector.

_aggregate_fov_result writes the seg array to the HCS plate (still in the
parent — concurrent create_position from workers would race plate
metadata), extends per-T row lists, and extends the 24 dataset-level
accumulators.

This refactor positions C4 to add ProcessPoolExecutor + worker primitives
without further changes to the FOV body. flush_manifest stays in the
parent's loop for now; under process mode (C4) each worker will flush
its own ctx at FOV end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds worker primitives in runtime.py (_worker_initializer,
gpu_serialization_lock, make_fov_executor) and the worker entry point
_worker_run_fov in pipeline.py. evaluate_predictions now branches on
runtime.executor: serial (default) runs _process_one_fov inline
identical to today; process spawns a ProcessPoolExecutor over FOVs.

Worker design:
- Spawn context avoids inheriting parent CUDA. _worker_initializer
  sets BLAS/OMP env caps BEFORE apply_thread_budget lazy-imports torch.
- _worker_setup lazy-loads seg_model + extractors under
  gpu_serialization_lock(): N workers don't init CUDA + load weights
  concurrently. Plate handles cached per-worker.
- GPU touch sites inside _process_one_fov are wrapped in
  gpu_serialization_lock(): mask_gt/pred, cp_gt, deep_gt_{dinov3,dynaclr,
  celldino}, features_pred_per_t, pixel_metrics (per-T, GPU-bound
  end-to-end including cubic spectral), and on-demand cellpose fallback
  inside mask_metrics. In serial mode the lock has no path set and yields
  immediately — bit-for-bit-identical behavior.
- Lock file at /tmp/dynacell_gpu_<SLURM_JOB_ID-or-PID>.lock (node-local,
  not NFS). One lock per SLURM job, shared across workers.

Phase-2 runtime resolution: after pred_positions is built,
resolve_runtime is re-called with n_positions= to clamp "auto"
fov_workers to min(provisional, n_positions) and freeze
threads_per_worker at the Phase-1 value so worker initializers match
the parent's already-applied BLAS cap.

Order: results collected via as_completed (arrival order) but aggregated
in original pred_positions order so per-FOV CSV + embedding NPZ row
order is deterministic regardless of executor.

Workers flush their own ctx manifest at FOV end (global manifest lock at
_pos_write_lock(ctx, "manifest", "global") fcntl serializes flushes
across workers); interrupted process-mode runs preserve cache progress
the same way serial mode did.

Tests: full dynacell suite 384/48 pass/fail (identical to baseline; all
48 failures pre-existing). No new spawn-mode test in this commit —
covered by C5 CPU integration test.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…docs (C5)

Adds test coverage for the C1-C4 runtime module and FOV-parallelism wiring:

  test_runtime.py (23 tests): apply_thread_budget no-crash semantics under
  Hydra's main; resolve_runtime auto/freeze/validate paths; region timing
  collector; DYNACELL_FORCE_PER_T_HYGIENE env-var hook; spawn-context
  ProcessPoolExecutor smoke via stdlib os.getpid.

  test_evaluation_pipeline_parallel.py (3 tests): FovResult pickle round-trip
  with realistic per-backbone shapes (the path that ProcessPoolExecutor takes
  for Future.result transport); _aggregate_fov_result extends all 24
  dataset-level accumulator lists correctly. Re-imports the live pipeline
  module inside test bodies because test_lazy_init.py pops dynacell.* modules
  from sys.modules, which would invalidate pickle resolution of a
  module-level imported FovResult.

CI wiring:
  - Root pyproject.toml registers the "slow" marker and appends -m "not slow"
    to existing addopts.
  - test-dynacell-configs runs test_runtime.py + test_evaluation_pipeline_parallel.py
    alongside existing benchmark-schema/submit-tool tests. No new heavy deps
    needed (these tests don't import iohub, cubic, transformers).

SLURM env caps:
  - run_eval_jointtrained_nucleus.slurm and
    run_eval_unext2_jointtrained_membrane.slurm export
    DYNACELL_THREADS_PER_WORKER=\$SLURM_CPUS_PER_TASK before invoking eval.

Docs:
  - applications/dynacell/CLAUDE.md gains "Eval runtime / parallelism"
    section documenting the config block, env-var hooks, timing
    instrumentation, and process-mode caveats.

Full dynacell test suite: 410 passed / 48 failed / 65 skipped — the 48
failures are the same set that fails on dynacell-models@HEAD (verified
parity across all 5 commits in this branch).

End-to-end serial-vs-process parity test on a real iohub+cache fixture is
a follow-up (see .claude/plans/eval-parallelism.md §C5).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds a runtime.is_worker() helper that returns True iff
_worker_initializer has set _GPU_LOCK_PATH. Wire it into
_process_one_fov's inner-T tqdm via disable=is_worker(), so under
fov_workers>1 the N concurrent per-T bars don't interleave on the
parent's stderr. Outer per-FOV tqdm (in the parent) stays visible
because the parent's _GPU_LOCK_PATH is never set.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Bug: _process_one_fov writes region timings to the global _TIMINGS
collector via region_timer, and returns the slice in FovResult.timings.
_aggregate_fov_result then calls extend_timings(result.timings)
unconditionally. In serial mode this re-appends each row to the SAME
parent collector — every timed region appears twice in eval_timing.csv.

In process mode the second append is correct (workers have separate
per-process _TIMINGS, parent's is empty until aggregated).

Fix: add extend_worker_timings kwarg to _aggregate_fov_result. Pass
True in process mode, False in serial. Verified via 63-test suite green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

_worker_setup no longer caches pred/gt/seg plate handles in _WORKER_STATE.
Instead, _worker_run_fov opens them via `with open_ome_zarr(...) as ...`
scoped to one FOV — iohub file descriptors close between FOVs and the
worker has nothing to clean up on ProcessPoolExecutor shutdown.

Models + cache contexts stay cached in _WORKER_STATE (their load cost
~5-10 s per worker is too high to repeat per FOV; ProcessPoolExecutor
process exit reclaims them).

Position lookup by name uses a linear scan of plate.positions() (iohub
has no name->position index). Cheap for the ≤ tens of positions per
typical eval — sub-ms per scan.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…_cache

When io.require_complete_cache=true, the caller asserts that all needed
masks live in the cache — so loading SuperModel (which hits a quilt
download for nucleus/membrane on a cold node) is wasted work. Return
None unconditionally in that mode. If a cache miss happens despite the
assertion, the cache layer raises StaleCacheError before any segment()
call, so the None seg_model never gets dereferenced.

Enables the cache-only/no-model path that C5's integration test design
relies on (run evaluate_predictions end-to-end against prebuilt mask
caches without instantiating cellpose/SuperModel).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Drives evaluate_predictions end-to-end on a tiny iohub fixture (3 positions
× T=2 × D=4 × H=W=32) with prebuilt mask caches. Runs the pipeline twice —
once with runtime.executor=serial, once with runtime.executor=process
runtime.fov_workers=2 — and asserts:

  1. pixel_metrics + mask_metrics rows are byte-equal after sort by FOV.
  2. segmentation_results.zarr per-position content is byte-equal across
     both channels.

Cache-only design (matches plan §C5): target_name=er +
require_complete_cache=true + compute_feature_metrics=false +
compute_microssim=false + pixel_metrics.fsc=null +
pixel_metrics.spectral_pcc=null. The full pipeline runs without
instantiating cellpose, SuperModel, DinoV3, DynaCLR, CellDino, or cubic.

What this validates: spawn boot, DictConfig pickling, FovResult.timings +
arrays round-trip through pickle, _aggregate_fov_result extends parent
accumulators correctly, parent-side segmentation_results.zarr writes
don't race, sort-by-pos_name aggregation, and the new
extend_worker_timings semantics doesn't double-count timing rows.

_live_pipeline_module() clears dynacell.evaluation.* from sys.modules
before importing — necessary because test_evaluation_pipeline.py stubs
metrics/utils via monkeypatch.setitem(sys.modules, ...), and without the
clear my pipeline's compute_pixel_metrics resolves to the empty-returning
stub.

CI: runs ~16 s on the existing test-dynacell-configs job.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds tests/test_evaluation_pipeline_parallel_cpu.py to the dynacell
pytest invocation. The test runs end-to-end evaluate_predictions in
both serial and spawn-process modes against a tiny iohub fixture +
prebuilt mask cache — no eval extras (cellpose, transformers, cubic)
needed because target_name=er + io.require_complete_cache=true
short-circuit segmenter and feature-extractor loads.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Four Copilot-review findings on the runtime layer:

1. apply_thread_budget threadpool_limits re-arm: the old handle becomes
   unreachable and is GC'd. The previous comment said "stays alive"
   which was misleading. Document the actual behavior: we WANT the new
   limit active, dropping the old handle is intentional. Also remove
   the dead if-branch since both arms did the same assignment.

2. resolve_runtime accepts literal fov_workers=0 / threads_per_worker=0
   which produces divide-by-zero in the "auto" math or nonsensical
   thread counts. Add explicit >= 1 validation with a clear error.

3. _cpu_count() prefers os.sched_getaffinity(0) over os.cpu_count() so
   SLURM cgroup limits (cpuset etc.) are honored — otherwise auto-
   resolved fov_workers/threads_per_worker can oversize past the
   cgroup-visible CPU budget. Falls back to os.cpu_count() on non-Linux.

4. gpu_serialization_lock now takes a gate: bool = True kwarg. Callers
   in pipeline.py pass gate=use_gpu so under config.use_gpu=false the
   lock is a no-op even in process mode. Previously the lock fired
   unconditionally, defeating cross-worker parallelism for CPU-only
   eval runs. The lock still gates on _GPU_LOCK_PATH for the
   parent/serial no-op (unchanged).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ak memory

Previously buffered every FovResult in results_by_pos until ALL N
worker futures completed, then aggregated in pos_names_in_order. Peak
parent memory: N × seg_array (~N × 440 MB for T=10 D=21 H=W=1024 evals)
all live simultaneously.

Switch to streaming aggregation: drain the buffer opportunistically in
pos_names_in_order as futures complete. Only out-of-order FovResults
stay buffered; once the next-expected position lands we aggregate
(plate write + accumulator extend) and drop the ref. Peak buffer is
bounded by the number of out-of-order FOVs in flight, not by N.

Deterministic per-FOV CSV / embedding NPZ row order is preserved
because aggregation still happens in pos_names_in_order — early
completions just wait their turn in the buffer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… cache off

`prepare_segmentation_model` was returning None unconditionally
whenever `io.require_complete_cache=true`, on the assumption that all
masks would be served from cache. That assumption breaks for
nucleus/membrane targets when `io.pred_cache_dir=None`: the per-T loop
in `_process_one_fov` falls back to `segment(predict[t],
seg_model=seg_model)`, which raises `ValueError` because `seg_model` is
None.

Narrow the early return to cases where no path can reach `segment()`
without a cached mask — organelle targets (workflow-based, no
`seg_model` needed) or nucleus/membrane runs that also have
`io.pred_cache_dir` configured. Reported by Copilot on PR #426.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…ation

The docstring claimed "no CUDA available in the worker" was one of the
no-op conditions, but the implementation only checks `gate` and
`_GPU_LOCK_PATH`. On a CPU-only node with `use_gpu=true` the lock would
still serialize across workers. Rather than add a runtime CUDA probe
(callers already pass `gate=use_gpu` to opt out for CPU-only regions),
correct the docstring to describe what the code actually does.
Reported by Copilot on PR #426.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…im bloat

The eval README had grown stale relative to the runtime/grouped/cache
work landed on this branch and PR #427: no mention of
`io.require_complete_cache`'s cache-only fast path, no `runtime.*`
block, no `dynacell evaluate-grouped`, no CELL-DINO.

This rewrite folds those in and restructures: FOV parallelism + grouped
eval move out of "Artifact caches" (where they were mis-classified)
into a new top-level "Scaling" section; a new "Quick start" sits above
Configuration; HF-cache operational details go to a new "HPC notes"
section at the end.

Bloat trim along the way: components table compressed (related modules
combined, trivial files pushed to a note), HF-cache + external-users
sections shortened, the 11-row force_recompute table collapsed to a
YAML block plus 5 grouped descriptions, redundant Hydra
group-discovery paragraphs removed. No flags, footguns, or CLI
options dropped — every config knob and env var from the old version
is still documented. 466 → 265 lines.

Public-repo hygiene: replaced the hard-coded
`/hpc/projects/comp.micro/...` default for the team-shared HF cache
with a description that points at the `_DEFAULT_SHARED_HF_CACHE`
constant in `dynacell/__main__.py` and the `DYNACELL_SHARED_HF_CACHE`
env-var override.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Five fixes from a code-review pass on the controlled FOV-parallel eval
work:

- Restore the seg_model invariant for grouped multi-condition runs.
  After 5829175, ``prepare_segmentation_model``'s load decision
  depends on ``io.pred_cache_dir`` in addition to
  ``io.require_complete_cache``, but ``_MODEL_LOADING_FIELDS`` only
  caught the latter. A per-condition overlay setting
  ``io.pred_cache_dir=None`` against a base that has it set (for
  nucleus/membrane + ``require_complete_cache=true``) would crash
  at runtime: the shared ``seg_model`` is None but the per-T loop
  falls back to ``segment(predict[t], seg_model=None)``. New
  ``_seg_model_required()`` helper mirrors the load decision and
  ``_check_grouped_field_invariants`` rejects the dangerous direction
  only, leaving canonical per-condition pred caches free to differ.

- Fix the process-mode progress bar stalling. ``pbar.update(1)`` only
  fired inside the in-order drain loop, so if FOV 0 was slow and later
  FOVs completed first, the bar stayed at 0% until FOV 0 drained.
  Update on each worker completion instead.

- Correct the misleading ``[grouped] note:`` for ``executor=process`` +
  ``require_complete_cache=true``. The message claimed "no models
  actually load" but deep extractors do load whenever
  ``compute_feature_metrics=true``, and the segmenter only short-
  circuits in a subset of cases. Rewrite to match reality.

- Move ``runtime.*`` imports (12 symbols) from four inline
  per-function blocks to one top-of-file import. No spawn-ordering or
  circular-import constraint blocks them; the inline pattern just
  violated the project's "no inline imports without strong reason"
  rule and duplicated a ``flush_manifest`` import that already lives
  at module top.

New regression test ``test_grouped_rejects_pred_cache_dir_flip_for_nucleus``
guards the dangerous direction; an existing test's assertion updates to
the corrected note wording.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>